Method and apparatus to extend the effective dynamic range of an image sensing device

ABSTRACT

An image sensor generates an image signal with a differential response to image light. The image sensor has an array of photosites divided into standard photosites and non-standard photosites. A limiter provides the standard photosites with a predetermined standard response to a light exposure and the non-standard photosites with a predetermined slower response to the same light exposure. The standard photosites and nonstandard photosites both sparsely sample the array in a predetermined pattern.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a divisional of U.S. application Ser. No. 10/946,919 filed Sep.22, 2004, which is a continuation-in-part of U.S. application Ser. No.09/615,398 filed Jul. 13, 2000, now U.S. Pat. No. 6,909,461 issued Jun.21, 2005. The disclosure of the priority applications, in theirentirety, including the drawings, claims, and the specification thereof,are incorporated herein by reference.

FIELD OF THE INVENTION

The invention relates generally to the field of image capture, and morespecifically to a method of extending the effective dynamic range of animage sensing device.

BACKGROUND OF THE INVENTION

Image sensing devices, such as a charge-coupled device (CCD), arecommonly found in such products as digital cameras, scanners, and videocameras. These image sensing devices have a very limited dynamic rangewhen compared to traditional negative film products. A typical imagesensing device has a dynamic range of about 5 stops. This means that theexposure for a typical scene must be determined with a fair amount ofaccuracy in order to avoid clipping the signal. In addition, oftentimesthe scene has a very wide dynamic range as a result of multipleilluminants (e.g. frontlit and backlit portions of a scene). In the caseof a wide dynamic range scene, choosing an appropriate exposure for thesubject often necessitates clipping data in another part of the image.Thus, the inferior dynamic range of an image sensing device relative tosilver halide media results in lower image quality for images obtainedby an image sensing device.

An increase in the dynamic range of an image sensing device would allowimages from digital cameras to be rebalanced to achieve a more pleasingrendition of the image. Also, increasing the dynamic range of an imagesensing device would allow for more pleasing contrast improvements tothe image, such as is described by Lee et al. in commonly assigned U.S.Pat. No. 5,012,333.

U.S. Pat. No. 6,040,858 (Ikeda) provides a complete description of theproblem of the limited dynamic range of image sensing devices. Inaddition, Ikeda describes methods of extending the dynamic range of animage sensing device by utilizing multiple image signals, each withdifferent responses to exposure. These multiple signals are combined byusing thresholds, which determine which signal is of higher quality ateach position in the image signal to form an image signal havingextended dynamic range. Ikeda improves upon these methods by describinga method by which these thresholds are determined for each color.

However, these prior art methods, including Ikeda, require multipleimage signals to form an image signal having extended dynamic range.Attaining such multiple signals can cause difficulty. For example, ifthe multiple image signals are not captured simultaneously, objectsmoving in the scene or motion of the camera may produce artifacts in animage signal having extended dynamic range. Additionally, if the imagesignals are captured simultaneously but on separate image capturedevices, then a correspondence problem exists. Moreover, the additionalhardware adds undesirable cost to the imaging system.

U.S. Pat. No. 5,051,770 to Cornuejols discloses a device, in which aflat screen modifies, on a pixel by pixel basis, light received by asensor.

U.S. Published Patent Application 2003/0052976 describes a photographicdevice having photosites of different sensitivities, controlled by drivevoltages. Different exposure times can be imparted to the photosites.

Thus, there exists a need to improve upon the method of the prior art inorder to improve the dynamic range of an image sensing device.Specifically, there exists a need to produce an extended dynamic rangeimage signal with a single image sensing device and a single imagesignal.

SUMMARY OF THE INVENTION

The invention is defined by the claims. The present invention isdirected to overcoming one or more of the problems set forth above.Briefly summarized, according to one aspect of the present invention, animage sensor generates an image signal with a differential response toimage light. The image sensor has an array of photosites divided intostandard photosites and non-standard photosites. A limiter provides thestandard photosites with a predetermined standard response to a lightexposure and the non-standard photosites with a predetermined slowerresponse to the same light exposure. The standard photosites andnonstandard photosites both sparsely sample the array in a predeterminedpattern.

In the present invention, selected photosites of the image capturedevice are designed to have non-standard response to exposure. Thesenon-standard photosites generally have a slower response to exposurethan do the non-selected, or standard, photosites. The advantage of theinvention is that the image signal from such an image capture device isprocessed to take advantage of the dynamic ranges of all photosites.Thus, an image signal having increased dynamic range is produced byinterpolating the values of neighboring photosites for those photositesthat are saturated or at a noise level.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned and other features and objects of this invention andthe manner of attaining them will become more apparent and the inventionitself will be better understood by reference to the followingdescription of an embodiment of the invention taken in conjunction withthe accompanying figures wherein:

FIG. 1 is a block diagram of an extended-range image sensing deviceaccording to the invention.

FIG. 2 is a graph illustrating the response of a standard photosite anda non-standard photosite.

FIG. 3A illustrates the arrangement of non-standard photosites andstandard photosites on a panchromatic embodiment of the image sensor.

FIG. 3B illustrates the arrangement of non-standard photosites andstandard photosites on an embodiment of a color image sensor.

FIG. 4 shows an exploded block diagram of the dynamic range extendingfilter array (DREFA) processor shown in FIG. 1.

FIG. 5 is a cross-section of an interline image sensor employing anarray of lenslets to alter the response of selected photosites.

FIG. 6 is a cross-section of a full frame image sensor employing a metalmask to alter the response of selected photosites.

FIG. 7 is a cross-section of an image sensor employing an array ofneutral density filters to alter the response of selected photosites.

FIG. 8 is a block diagram of an embodiment of the image capture system.

FIGS. 9A-9C illustrate the arrangement of non-standard photosites andstandard photosites in three different embodiments of the image sensor

FIG. 10 illustrates the arrangement of non-standard photosites andstandard photosites on another embodiment of a color image sensor.

FIG. 11 illustrates the same embodiment as FIG. 10 and shows an 8×8array having a 4×4 repeating unit.

FIG. 12 illustrates an 8×8 array of another embodiment of the colorimage sensor having an 8×8 repeating unit.

FIG. 13 illustrates the same embodiment as FIG. 10 and shows a 16×16array having a 4×4 repeating unit.

FIG. 14 illustrates another embodiment of the color image sensor havinga 16×16 repeating unit.

FIG. 15 illustrates the layout of components of a prior art device thatcan be used in another embodiment of the color image sensor.

DETAILED DESCRIPTION OF THE INVENTION

Because imaging devices employing electronic sensors are well known, thepresent description will be directed in particular to elements formingpart of, or cooperating more directly with, apparatus in accordance withthe present invention. Elements not specifically shown or describedherein may be selected from those known in the art. Note that as usedherein, the term image is a two dimensional array of values. An imagemay be a two dimensional subset of another image.

Referring to FIG. 1, light from an object or scene is incident upon alens 2. An optical low pass filter 6 performs a slight blurring of theimage in order to reduce the occurrence of aliasing. The image falls onan image sensing device 10 such as a charged-coupled device (CCD). Notethat other devices, such as CMOS devices, may be used as the imagesensing device 10.

An A/D converter 14 converts the image signal from the image sensingdevice 10 into a digital signal. More specifically, the A/D converter 14converts the linear voltage signal from the image sensing device 10 to adiscrete digital signal, preferably a 10 bit signal. Thus, the linearencoded values range from 0 to 1023. The A/D converter 14 alsopreferably performs processing to convert the linear 10 bit signal to an8 bit logarithmic signal, as is commonly performed in the art. Thefollowing equation is used to convert the 10 bit linear signal a(x,y),where (x,y) specifies the row and column index of the signal locationwith reference to the image sensing device 10, into the 8 bitlogarithmic signal b(x,y):

${b\left( {x,y} \right)} = \left\{ \begin{matrix}0 & {0 \leq {a\left( {x,y} \right)} \leq 31} \\{{73.5975\mspace{11mu}\ln\;{a\left( {x,y} \right)}} - 255} & {32 \leq {a\left( {x,y} \right)} \leq 1024}\end{matrix} \right.$Note that each stop of exposure (in the linear response region of theimage sensing device) results in a doubling of the linear signal a(x,y)and results in an increase of the logarithmically encoded signal b(x,y)by 51. In this case, the value 51 represents the number of code valuesper stop (cvs) of exposure.

The system controller 18 determines on the basis of user input or on thebasis of the image signal output from the A/D converter 14 whether thedynamic range needs to be expanded (i.e. with the “expanding mode” ofprocessing) or whether there is no need for expanding (i.e., the “normalmode” of processing). The system controller 18 then diverts the digitalimage signal b(x,y) to either the dynamic range extending filter array(DREFA) processor 22 if the system controller 18 is in an expandingmode, or to the color filter array (CFA) interpolator 26 if the systemcontroller 18 is in a normal mode. Alternatively, the system controller18 may be set at the time of manufacture to always be in the “expandingmode.”

In the “normal mode”, the system controller 18 diverts the image signaloutput from the A/D converter 14 to the CFA interpolator 26. The purposeof the CFA interpolator 26 is to generate a full description of thecolor for each location of the digital image. In the preferredembodiment, the image sensing device 10 consists of an array ofphotosensitive elements. Each photosite is typically coated with eithera red, green, or blue filter, as described by Bayer in commonly assignedU.S. Pat. No. 3,971,065, which is incorporated herein by reference. TheBayer array is a color filter array in which green filters are locatedin a checkerboard pattern over the photosites with red and blue filteralternating line by line to fill the interstices of the checkerboardpattern; this produces twice as many green filter sites as either red orblue filter sites. Note that the method described herein may be easilyextended to color filter arrays with different arrangements of theprimaries, a different number of primaries, or a different set ofprimaries. Thus, in the preferred embodiment, each photosite issensitive to either red, green, or blue light. However, it is desirableto obtain a value of exposure for each of the red, green, and blueexposures at each photosite location. In this description, “red”,“green”, and “blue” represent the primaries of an image sensing device10, as is well known in the art of image processing. A CFA interpolator26 generates from the image signal output from the A/D converter 14 aninterpolated image signal consisting of a value for each of theprimaries of a photosensitive element. For example, if a particularphotosite is coated with a red filter, then the A/D converter 14 outputsa red level of exposure for that photosite since the red filteressentially blocks green and blue light from reaching the image sensingdevice 10.

The operation of the CFA interpolator 26 is to determine the levels ofexposure for a red photosite for both the green and the blue primaries.Similarly, the CFA interpolator 26 determines the green and red exposurelevels for the blue photosites, as well as the red and the blue exposurelevels for the green photosites. Generally, the CFA interpolator 26operates by considering the exposure values of the photosite and thevalues of surrounding photosites. While any commonly known interpolatormay be used, a description of a preferred CFA interpolator is containedin commonly assigned U.S. Pat. No. 5,652,621, entitled “Adaptive colorplane interpolation in single sensor color electronic camera”, which isincorporated herein by reference. This patent describes apparatus forprocessing a digitized image signal obtained from an image sensor havingcolor photosites aligned in rows and columns that generate at leastthree separate color values but only one color value for each photositelocation, and a structure for interpolating color values for eachphotosite location so that it has three different color values. Theapparatus generates an appropriate color value missing from a photositelocation by the interpolation of an additional color value for suchphotosite locations from color values of different colors than themissing color value at nearby photosite locations. The apparatus alsoobtains Laplacian second-order values, gradient values and colordifference bias values in at least two image directions from nearbyphotosites of the same column and row and selects a preferredorientation for the interpolation of the missing color value based upona classifier developed from these values. Finally, the missing colorvalue from nearby multiple color values is selected to agree with thepreferred orientation.

In the “expanding mode”, the system controller 18 diverts the imagesignal output from the A/D converter 14 to the DREFA processor 22 inorder to expand the dynamic range of the image signal. In the preferredembodiment, the dynamic range of the image sensing device 10 is expandedby selecting certain photosites of the image sensing device 10 to have anon-standard response. The arrangement of the selected photosites withrespect to the image sensing device 10 will be discussed in greaterdetail hereinbelow. In the preferred embodiment, the responses ofselected photosites are slowed by altering the gain of the selectedphotosites, herein referred to as non-standard photosites. Altering thegain of a photosite is commonly practiced in the art of digital cameradesign and manufacture.

With reference to FIG. 5, it is a common practice in the art of imagesensor manufacture to place resin lenslets 51 on top of each photosite.For example, particularly when the image sensing device 10 is aninterline solid state image sensing device, one lenslet technique isdescribed in U.S. Pat. No. 4,667,092, entitled “Solid-state image devicewith resin lens and resin contact layer”, which is incorporated hereinby reference. In this patent, more specifically, a solid-state imagedevice includes an image storage block having a block surface and aplurality of storage elements are embedded along the block surface tostore an image in the form of electric charge. An overlying layer isdeposited to form an array of optical lenses in correspondence to thestorage elements. An intermediate layer is laid between the blocksurface and the overlying layer. Incident light focuses through thelenses and the intermediate layer onto the storage elements. Theintermediate layer serves as an adjusting layer for adjusting a focallength.

FIG. 5 shows a cross section of an interline solid state image sensingdevice. Without the lenslets 51, the signal readout area associated witheach photosensitive area 55 of a photosite makes it impossible to usethe whole area of the semiconductor substrate as the photoelectrictransducer area. The conventional solid-state image device does noteffectively utilize all incident rays thereon and therefore has lowsensitivity. The addition of a resin lenslet 51 on top of a photositeallows the incident rays of light to be focused on the photoactive areasof the photosite, thereby more effectively utilizing the incident raysof light and increasing the sensitivity of the photosite. Thus, byvarying the size and/or efficiency of the lenslet 51, the sensitivity(gain) of the photosite may be easily altered. Thus, for interlinedevices and for CMOS sensors, the preferred method of altering the gainof the photosite is by altering the lenslet 51 placed on top of thephotosite. As shown in FIG. 5, the location 52 has no lenslet, andtherefore fewer incident rays of light are incident with thephotosensitive area. Alternatively, a lenslet could be manufactured atlocation 52 with a different radius, shape, size or material as comparedwith the lenslet 51, thereby structured to be less efficient at focusingincident rays of light onto the photosensitive area 55 than is thestandard lenslet 51. Those skilled in the art will recognize that if thelenslet 51 focuses 80% of the incident rays of light onto aphotosensitive area 55 and the region 52 having no lenslets (oralternatively non-standard lenslets) allows 20% of the incident rays oflight onto a photosensitive area 55, then the photosite covered bylenslet 51 is 2 stops faster than the region 52. In this case, thelenslet 51 is used for standard photosites and no lenslet is used fornon-standard photosites, as represented by region 52.

With reference to FIG. 6 showing a cross section of a full frame imagesensing device 10, in the case where the image sensing device 10 is afull frame device, light rays incident to the photosensitive area 55 ofa photosite must pass through an aperture of a mask, typically made frommetal, which is shown in cross-section in FIG. 6 to compriselight-blocking metallic mask portions 54 and open apertures 56 and 57interspersed among the metallic portions. In the preferred embodiment,the gain of photosites may be altered by modifying the metal mask 54light shield. The sensitivity of the photosite is then directly relatedto the aperture of the metal mask 54 light shield. For example, onephotosite with an aperture 50% of the size of a second photositesaperture will have a response of 50% compared to that on the secondphotosite. For example, a first aperture 56 of a light shield 54 allows80% of the light rays incident upon that photosite to pass through, buta second aperture 57 is smaller and allows only 20% of the incidentlight rays to pass. Those skilled in the art will recognize that thephotosite with the larger first aperture 56 is 2 stops faster than aphotosite with the smaller second aperture 57. In this case, the firstaperture 56 is used for standard photosites, and the second aperture 57is used for the non-standard photosites. Thus, the aperture of a lightmask may be modified to adjust the response of the selected photosites.Kodak makes full frame image sensing devices with a metal mask lightshield that reduces the pixel active area of all pixels from about 80%to about 20% (for dithered scanner applications where the sensor ismoved by ½ the pixel spacing horizontally and vertical and 4 picturesare taken). The invention thus involves utilizing such mask technology,but with different sized apertures, to provide an image sensor with adifferential response to image light.

Electronic control of the individual photosites can also provide thestandard photosites and non-standard photosites. For example, U.S.Patent application 20030052976 describes a photographic device havingphotosites of different sensitivities, controlled by drive voltages.Different exposure times can be imparted to the photosites via the drivevoltages. For example, the photosites having the shortest exposure timesare non-standard photosites and photosites having longer exposure timesare standard photosites.

FIG. 15 shows an example of an image sensor having photosites withdifferent sensitivities (i.e. standard photosites and non-standardphotosites) by using drive voltages to control the sensitivities of thephotosites, as described in U.S. Published Patent Application2003/0052976. FIG. 15 illustrates a plurality of photosites 208 a to 208h, a plurality of vertical transfer registers 207 a and 207 b a firstdrive-voltage applying electrode 203, and a second drive-voltageapplying electrodes 201 and 202. The photosites (also called PDs orphotodiodes) 208 a to 208 h are arranged in a horizontal direction and avertical direction. The vertical transfer registers 207 a and 207 btransfer the electric charges accumulated in the photosites in thevertical direction. The first drive-voltage applying electrode 203 isarranged parallel to the vertical transfer registers, for applying adrive voltage to a specific one of the vertical transfer registers. Thesecond drive-voltage applying electrode 201 to 202 is arrangedperpendicular to the vertical transfer registers, for applying a seconddrive voltage to the vertical transfer registers at the same time. Theelectric charges accumulated in the photosites are transferred to thevertical transfer registers, due to the voltage output from the firstdrive-voltage applying electrode or the second drive-voltage applyingelectrode, or the voltages output from both electrodes. Therefore, thephotosites can have different sensitivities. The photosites with thestandard response to a light exposure are called standard photosites andthe photosites with a slower response to the same light exposure arecalled non-standard photosites.

In the preferred embodiment, the response of the selected non-standardphotosites is X % (where X<=100) that of standard photosites for thesame exposure, as shown graphically in FIG. 2. The selected photositeshave a response that is slowed by two stops (−log X/100) relative to thestandard photosites. In the preferred embodiment, X=25. The value ofparameter X (i.e. the amount of difference between the response of thestandard photosites and the non-standard photosites) effectively limitsthe dynamic range of the image sensor. The value of parameter X can be aconstant set at the time of manufacture, or selected by the user priorto image capture or selected by an algorithm residing in the systemcontroller 18 that analyzes image data from a prior image of the sceneto determine the dynamic range of the scene, then sets the value ofparameter X appropriately. In the preferred embodiment, when the systemcontroller 18 is in “normal mode”, all photosites have common gainequivalent to that of the standard response. Thus, the image sensingdevice 10 consists of multiple sets of photosites, the standardphotosites and the non-standard photosites. The collection of theoutputs of the standard photosites constitutes a sparsely sampledversion of a scene. Likewise, the collection of the outputs of thenon-standard photosites constitutes another sparsely sampled version ofa scene.

As another alternative, the responses of the selected non-standardphotosites can be slowed by the use of a neutral filter coating thephotosite. FIG. 7 shows a cross section of an image sensing device witha color filter array 53. Note that the color filter array 53 a is red,53 b is green, 53 c is red, and 53 d is green. A layer of neutralfilters 58 is contained above the color filter array 53, although theposition of the layer of neutral filters 58 and the color filter array53 does not matter. Note that the layer of neutral filters 58 onlycontains a neutral filter at the positions of selected photosites, asindicated by the neutral filter 59. In this case, the layer of neutralfilters 58 is transparent or nearly transparent for standard photositesand contains a neutral filter 59 for non-standard photosites. Forexample, if the neutral filter 59 consists of a material that allows X %transmission of light, then the response of that non-standard photositewill be slowed by

$- {\log_{2}\left( \frac{X}{100} \right)}$stops relative to the response of the standard photosite. If a neutralfilter 59 is used to create non-standard photosites, then the systemcontroller 18 is preferably set at the time of manufacture to be in“expanding mode.”

Implementation of the standard photosites and non-standard photosites isaccomplished by a limiter. The limiter is a physical or logical devicethat modifies sensitivity of a set of non-standard photosites on animage sensor such that the standard photosites have a firstpredetermined response to a light exposure and the non-standardphotosites have a second, slower predetermined response to the samelight exposure. The limiter can be implemented with a structural elementoverlaying the photosites such as the differing size lenslets overstandard and non-standard photosites, differing size apertures in themask for standard and non-standard photosites, and differing neutralfilters over standard and non-standard photosites. The limiter can alsobe a programmed feature of the image sensor and supporting electronics,such as a logical device that uses different drive voltages to controlexposure time and/or gain of the standard and non-standard photosites.

The purpose of the DREFA processor 22 is to create a digital imagesignal with an increased dynamic range by processing the digital imagesignal while considering the standard and non-standard photosites. Theoperation of the DREFA processor 22 will be described in detailhereinbelow. Accordingly, the output of the DREFA processor 22 is anexpanded image signal having increased dynamic range. This expandedimage signal is then input to the CFA interpolator for processing aspreviously described.

Note that although FIG. 1 implies that the A/D converter 14 and theDREFA processor 22 are directly connected, this is not a requirement forthe present invention. The DREFA processor 22 may reside in hardware orsoftware in close proximity to the A/D converter 14 and image sensingdevice 10. For example, the DREFA processor 22 could reside directlywithin a digital camera. However, the DREFA processor 22 may also beremote from the image sensing device 10. For example, referring to FIG.8, the image signal output from the A/D converter 14 can be transmitted(after compression) from the digital camera 100 to a host computer 104.Transmission can, optionally, be through a network 102. Likewise, theimage signal output from the A/D converter 14 can be transmitted (aftercompression) via a wire or wireless connection to a personal computingdevice, printer, or remote server (not shown) to apply to operation ofthe DREFA processor 22. Transmission of the image signal may alsoinclude file transfer protocol or email. Additionally, payment viacredit card or some other means may be required by the DREFA processor22 from the user.

The ratio of standard photosites to non-standard photosites rangesbetween 1:16 and 16:1 (numeric ranges herein are inclusive). The ends ofthis range are very sparsely sampled with one of the types of photositesresulting in low resolution for that type of photosite. A range withbetter resolution is between 1:4 and 4:1. In the preferred embodiment,50% of the photosites of the image sensing device 10 are selected tohave non-standard response. Those skilled in the art will recognize thatvarying the percentage of photosites that have non-standard responsewill still result in the advantages of the present invention. In thecase of a image sensing device 10 in which all photosites haveapproximately equivalent spectral sensitivity (i.e. a pan-chromaticimage sensing device), FIG. 3A shows an arrangement of the non-standardphotosites that will result in approximately 50% of all the photositesof the image sensing device 10 being of non-standard response. Thephotosites 28 with non-standard response are marked with an asterisk(*), while the photosites 30 having standard response are blank.

In the case of a color image sensing device 10, FIG. 3B shows anarrangement wherein 50% of each type (red, green, or blue sensitive) ofphotosite has non-standard response. For example, the photosite 32 is ared photosite having non-standard response, the photosite 34 is a greenphotosite having non-standard response, the photosite 36 is a bluephotosite having non-standard response, the photosite 38 is a redphotosite having standard response, the photosite 40 is a greenphotosite having standard response, the photosite 42 is a blue photositehaving standard response.

The 4×4 pattern shown in FIG. 3B can be repetitively tiled to fill animage sensor. In other words, the image sensor is made up of a repeatingpattern having a repeating unit of four by four pixels. It has beenanalytically determined that there are 648 patterns of 4 by 4 pixelswith the Bayer color filter array wherein 50% of each type (red, green,or blue sensitive) of photosite has non-standard response (of which FIG.3B is one example). The repeating unit can be larger than four by fourpixels as shown by FIGS. 12 and 14. In FIG. 12, an eight by eight pixelrepeating unit is shown. In FIG. 14, a sixteen by sixteen pixelrepeating unit is shown.

Note that FIGS. 3A and 3B imply a regular pattern for the location ofthe non-standard photosites. While it is preferable that thenon-standard photosites are arranged in a regular pattern, it is by nomeans necessary. The non-standard photosites could be arranged randomlyor semi-randomly over the surface of the image sensing device 10, andtheir location would be stored in some place accessible to the DREFAprocessor 22.

The response of a standard photosite to a certain exposure and theresponse of a non-standard photosite to the same exposure are shown inFIG. 2. Note that if a level of noise n is superimposed on the response,it can easily be seen that the standard photosite will yield a validsignal with lower exposures (beginning at exposure level E) than willthe non-standard photosite (which yields valid signal beginning at

$\frac{100}{X}{E.\text{)}}$Alternatively, data from the non-standard photosite will be valid forhigher exposure levels (up to signal level of

${\frac{100}{X}E\; 2^{s}},$where S is the inherent dynamic range of a single photosite, typicallyS=5 stops) than would the standard photosite (which produces validresponse up to an exposure of E2^(S).) Note that both the standardphotosite and the non-standard photosite have the same range of responsein stops of exposure (S), but the response of the non-standardphotosites is preferably

$- {\log_{2}\left( \frac{X}{100} \right)}$stops slower than the standard photosites, as shown in FIG. 2. It ispreferred that the responses of the standard and non-standard photositesoverlap with respect to exposure. That is, it is preferred that

${- {\log_{2}\left( \frac{X}{100} \right)}} < {S.}$The overall dynamic range of the image sensing device, considering bothstandard and non-standard photosites, is

$S - {{\log_{2}\left( \frac{X}{100} \right)}.}$In the case of the preferred embodiment, where S=5 and X=25, the overalleffective dynamic range of the image sensing device 10 is 7 stops ofexposure.

The processing of the DREFA processor 22 may be utilized to extend theoverall dynamic range of the image sensing device 10 by using thenon-standard photosite response to reconstruct the areas in the imagewhere very high exposures occur. Likewise, the DREFA processor 22 alsouses the photosites with standard response to reconstruct the signalwhere very low exposures occur.

FIG. 4 shows an exploded block diagram of the DREFA processor 22. Thelogarithmic image signal b(x,y) output from the A/D converter 14 ispassed to the non-standard compensator 44. The purpose of thenon-standard compensator 44 is to compensate the non-standard photositesby accounting for the offset in response by X stops. In the preferredembodiment, the image signal corresponding to the non-standardphotosites are incremented by the quantity−cvs log(X/100),where cvs is the number of code values per stop of exposure. In thepreferred embodiment, the quantity cvs is 51.

Alternatively, if the image signal input to the non-standard compensator44 is linearly related to exposure (rather than logarithmically), thenthe non-standard compensator 44 scales the image signal corresponding tothe non-standard photosites by a factor of 100/X. Note that it isassumed that the locations of the non-standard photosites are known tothe non-standard compensator 44. The output of the non-standardcompensator 44 is an image signal i(x,y) that has been compensated atthe locations of non-standard photosites for the difference between thenon-standard photosite response in relation to the standard photositeresponse. At the locations of standard photosites, the value of theimage signal b(x,y) output from the A/D converter 14 is identical to thevalue of the image signal i(x,y) output from the non-standardcompensator 44. Note that the image signal i(x,y) is not limited to an 8bit range. It the preferred embodiment, the value of i(x,y) ranges from0 to 357.

Next, the image signal i(x,y) output from the non-standard compensator44 is input to a non-standard thresholder 46. The purpose of thenon-standard thresholder 46 is to determine problem locations of thenon-standard image signal that are of low quality due to a photosite notreceiving enough photons to produce a valid signal or due to a photositereceiving so many photons that the signal saturates. The image signal atthese (x,y) locations is then replaced by calculating a new signal basedupon nearby samples of the standard image signal in processing performedby the signal extender 50, which will be described in detailhereinbelow. All (x,y) locations of the non-standard signal for whichthe corresponding values of the non-standard signal are less than apredetermined threshold are considered to be problem locations. In thecase of the non-standard photosite, this predetermined threshold usedfor the purpose of detecting problem locations is referred to as the lowexposure response threshold. Thus, a location (x,y) is considered to bea problem location if it is a non-standard photosite and if:i(x,y)<T ₁where T₁ is predetermined. In the preferred embodiment, the value of T₁is

${{- {cvs}}\;{\log_{2}\left( \frac{X}{100} \right)}},$which in the preferred embodiment is 102. Note that the threshold T₁ maybe dependent upon the color of the photosite at location (x,y).Non-standard photosites that are problem locations are referred to asnoise pixels, since the value of i(x,y) is not sufficiently about thenoise level of the image sensing device to be useful.

Likewise, the image signal i(x,y) output from the non-standardcompensator 44 is input to a standard thresholder 48. The purpose of astandard thresholder 48 is to determine problem locations of thestandard image signal that are of low quality. The image signal at theselocations is then replaced by calculating a new signal based upon nearbysamples of the non-standard image signal in processing performed by thesignal extender 50, which will be described in detail hereinbelow. All(x,y) locations of the standard image signal for which the correspondingvalues of the standard signal are less than a predetermined thresholdsignal are considered to be problem locations. In the case of thestandard photosite, this predetermined threshold used for the purpose ofdetecting problem locations is referred to as the high exposure responsethreshold. Thus, a location (x,y) is considered to be a problem locationif it is a standard photosite and if:i(x,y)>T ₂where T₂ is predetermined. In the preferred embodiment, the value of T₂is 254. Note that the threshold T₂ may be dependent upon the color ofthe photosite at location (x,y). Standard photosites that are problemlocations are referred to as saturated pixels, since the value of i(x,y)is as high as possible at these locations.

The problem locations determined by the non-standard thresholder 46 andthe problem locations determined by the standard thresholder 48 areinput to the signal extender 50. In addition, the image signal i(x,y)output from the non-standard compensator 44 is also input to the signalextender 50. The purpose of the signal extender 50 is to replace theimage signal i(x,y) values at problem locations (x,y) with estimates ofthe signal herein referred to as replacement values, had the inherentdynamic range of each photosite of the image sensing device 10 beengreater. If the problem location is coincident with a non-standardphotosite, then the replacement value is calculated from neighboringimage signal values coincident with standard photosites. In thisembodiment, the term “neighboring” refers to a certain spatial distance.In the preferred embodiment, the photosites neighboring a selectedphotosites are those photosites within a distance of 2 photosites of theselected photosite. Likewise, if the problem location is coincident witha standard photosite, then the replacement value is calculated fromneighboring image signal values coincident with non-standard photosites.In the preferred embodiment, the color of the photosite at the problemphotosite is also taken into account. The replacement value for anyproblem location is preferably determined only by the signal originatingfrom neighboring photosites of the same color. The output of the signalextender 50 is an image signal i′(x,y) having a dynamic range as ifcaptured by an image sensing device 10 having photosites with inherentdynamic range of

$S - {\log_{2}\left( \frac{X}{100} \right)}$rather than the actual inherent dynamic range of S for each photosite ofthe image sensing device 10. Note that for all (x,y) locations that arenot problem locations, the value of i′(x,y) is equivalent to i(x,y).

As an example of the processing performed by the signal extender 50 forthe Bayer CFA pattern shown in FIG. 3B, if location (x,y) is a problemlocation and (x,y) is the location of a green photosite (such asphotosite 34 in FIG. 3B), then the replacement value i′(x,y) for theimage signal i(x,y) is calculated in the following manner:i′(x,y)=0.25*[i(x−1,y−1)+i(x+1,y−1)+i(x−1,y+1)+i(x+1,y+1)]Note that signal values that the calculation of i′(x,y) is dependentupon are expected to comply with certain requirements. For example,suppose that (x,y) is a problem location and (x,y) is a green photositewith non-standard response. Then the signal levels of neighboringphotosites are used to calculate replacement value i′(x,y). However,this assumes that the signal values of each of the neighboringphotosites are also less than T₃. In the preferred embodiment, T₃=T₁.For each neighboring photosite that this is not the case, that signallevel is omitted from the calculation of the replacement value i′(x,y).For example, if i(x−1, y−1)>T₃, then the value i′(x,y) is calculatedwith the following formula:i′(x,y)=⅓*[i(x+1,y−1)+i(x−1,y+1)+i(x+1,y+1)]Generally, the interpolation scheme for determining a replacement valueat problem location (x,y), where the location (x,y) corresponds to agreen photosite which is also a standard photosite on a image sensingdevice having a Bayer pattern filter array is given with the followingequation:

${i^{\prime}\left( {x,y} \right)} = \frac{\sum\limits_{{j = {- 1}},1}{\sum\limits_{{k = {- 1}},1}{{i\left( {{x + j},{y + k}} \right)}{W\left( {{x + j},{y + k}} \right)}}}}{\sum\limits_{{j = {- 1}},1}{\sum\limits_{{k = {- 1}},1}{W\left( {{x + j},{y + k}} \right)}}}$where ${W\left( {{x + j},{y + k}} \right)} = \left\{ \begin{matrix}1 & {{i\left( {{x + j},{y + k}} \right)} > T_{3}} \\0 & {otherwise}\end{matrix} \right.$Note that the same equation is applied to determine the replacementvalue if the problem location corresponds with a green photosite whichis also a non-standard photosite. However, in this case

${W\left( {{x + j},{y + k}} \right)} = \left\{ \begin{matrix}1 & {{i\left( {{x + j},{y + k}} \right)} < T_{4}} \\0 & {{{otherwise},}\;}\end{matrix} \right.$where in the preferred embodiment, T₄=T₂.

As another example, also in connection with the Bayer CFA pattern shownin FIG. 3B, if location i(x,y) is a problem photosite and (x,y) is thelocation of a red or blue photosite, then the replacement value i′(x,y)for the image signal i(x,y) is calculated in the following manner:i′(x,y)=0.25*[i(x−2,y)+i(x+2,y)+i(x,y+2)+i(x,y−2)]When location (x,y) is a red or blue photosite and is also a standardphotosite, the equation for determining the replacement value i′(x,y)may be generalized as follows:

${i^{\prime}\left( {x,y} \right)} = \frac{\sum\limits_{{j = {- 2}},0,2}{\sum\limits_{{k = {- 2}},0,2}{{i\left( {{x + j},{y + k}} \right)}{W\left( {{x + j},{y + k}} \right)}}}}{\sum\limits_{{j = {- 2}},0,2}{\sum\limits_{{k = {- 2}},0,2}{W\left( {{x + j},{y + k}} \right)}}}$where ${W\left( {{x + j},{y + k}} \right)} = \left\{ \begin{matrix}1 & {{i\left( {{x + j},{y + k}} \right)} > T_{3}} \\0 & {otherwise}\end{matrix} \right.$Note that in this case, either j or k must be 0, but j and k are neverboth zero. Note also that the same equation is applied to determine thereplacement value if the problem location corresponds with a red or bluephotosite which is also a non-standard photosite. However, in this case

${W\left( {{x + j},{y + k}} \right)} = \left\{ \begin{matrix}1 & {{i\left( {{x + j},{y + k}} \right)} < T_{4}} \\0 & {{{otherwise},}\;}\end{matrix} \right.$where in the preferred embodiment, T₄=T₂.

The interpolation scheme described above for the purpose of generatingan image signal with an extended dynamic range from more than onesparsely sampled image signals may be modified by those skilled in theart. However, many such modifications to the above interpolation schememay be derived and should not be considered as significant deviations ofthe present invention.

Those skilled in the art will recognize that the above interpolationscheme performed by the signal extender is a lowpass filter, which iswell known in the art. Typically, the application of a lowpass filter toa digital image signal has a similar effect to reducing the resolutionof the digital image signal. Thus, the processing performed by the DREFAprocessor 22 is a method by which the spatial resolution of the imagesensing device 10 may be traded for dynamic range of the image sensingdevice 10. Indeed, those areas of an image where the interpolationscheme is implemented to increase the dynamic range of the signal appearnoticeably softer (less sharp) than the image would have if that samearea of the image had been captured by the sensor in such a fashion thatno “problem locations” (as defined by the non-standard thresholder 46and the standard thresholder 48) occur.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention. For example, the invention provides an image capturesystem that expands the dynamic range in both directions, i.e., thatexpands the response of the standard photosites to increased exposuresby utilizing the image signals from neighboring non-standard photositesand expands the response of the non-standard photosites to decreasedexposures by utilizing the image signals from neighboring standardphotosites. It would be likewise feasible for the system to work ondynamic range from only one direction, i.e., to expand the response ofonly the standard photosites to increased light exposures by utilizingthe image signals from neighboring non-standard photosites, oralternatively, to expand the response of only the non-standardphotosites to decreased light exposures by utilizing the image signalsfrom neighboring standard photosites.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   2 lens-   6 optical lowpass filter-   10 image sensing device-   14 A/D converter-   18 system controller-   22 DREFA processor-   26 CFA interpolator-   28 non-standard photosite-   30 standard photosite-   32 red non-standard photosite-   34 green non-standard photosite-   36 blue non-standard photosite-   38 red standard photosite-   40 green standard photosite-   42 blue standard photosite-   44 non-standard compensator-   46 non-standard thresholder-   48 standard thresholder-   50 signal extender-   51 standard lenslet-   52 non-standard lenslet-   53 color filter array-   54 metallic mask portion-   55 photosensitive area-   56 large aperture-   57 small aperture-   58 neutral density filter layer-   59 non-standard neutral density filter

1. An image capture system comprising: an image sensor including anarray of photosites divided into standard photosites and non-standardphotosites, wherein said standard photosites and said nonstandardphotosites both sparsely sample said array in a predetermined pattern;and a system controller having a normal mode and an expanding mode, saidsystem controller, in said normal mode, operating the image sensor withthe standard photosites and the non-standard photosites having a similarresponse to the same light exposure, and, in said expanding mode,operating the image sensor with the standard photosites having a firstresponse to a light exposure and the non-standard photosites having asecond, slower response to the same light exposure.
 2. The system ofclaim 1 wherein the system controller switches between said modes basedupon user input.
 3. The system of claim 1 wherein the system controlleris capable of adjusting relative magnitudes of said first and secondresponses based upon user input.
 4. The system of claim 1 wherein thesystem controller switches between said modes based upon analysis of animage signal captured by said image sensor.
 5. The system of claim 1wherein the system controller adjusts relative magnitudes of said firstand second responses based upon analysis of an image signal captured bysaid image sensor.